Python training array too large

0So I have over 30 million objects that I need to use as my training data. My Issue is simple: When I create my training array by an iterative appending process, at a certain threshold, the list becomes too large and python gets killed. What is a way to get around this? I have been trying to figure this out for hours and keep coming up short!

Code example for creating training array

training_array = []
for ...:
    data = #load data from somewhere
    data_array = [x for x in data] #some large array, 2-3 million objects  
    for item in data_array:
        training_array.append(item.a + item.b)

Python 02-05-22, 3:07 p.m. jagath


We are living in the era of digitalization where every brand wants to bring its business to the top of others in online business competition. Bring business online let's a brand to showcase its products and services. This is something that is only possible when you have a strong online presence which can be formed with a website only and this is something that leads the increasing demand for web designers.

At ADMEC, which is a multimedia institute and offers many web design courses in Delhi, I learned a lot. My program covers basic to advanced level concepts with good career opportunities.


These courses are developed by experts who have years of experience in the web industry. Y

ou can pursue diploma in both web design and development and even certificate programs are available. With live project based training and industry standard practical classes, you can pursue a successful career in this dynamic and rapidly growing industry.

Oops! Looks like spam! Waiting for moderator approval.
03-05-22, 3:44 p.m. nehaadmec

Login to add comment
Oops! Looks like spam! Waiting for moderator approval

Log-in to answer to this question.