The memory of loading our first substantial stock listings in 2010 is still clear. It was certainly not a painless procedure by any measurement. Our newly developed platform was not without it’s teething troubles and speed was certainly an issue, as well as capacity. The word “clunky” seems to describe it best, and initially it suffered from over-engineering ( down to the design team, which was mostly, errr , your editor) and this really slowed down the processing of searches and updates.

In addition, those first few listings took forever to convert and prepare, and then almost as long to load. We had to split them down into manageable chunks, 1,000 line items at a time, to get them to behave and our “anti-hijack” precautions seemed to bite us back at every opportunity.

The learning curve was steep and bit-by-bit it became easier to deal with “dirty” listings – those with no standardisation of syntax, content or even field layout. With practice we got quicker, the job became easier and the code became cleaner, but it was not until we commenced our major back-end upgrade in 2018, that the loading procedure really started to behave for us.

By the time we got to the point, in June 2021, where we made the decision to promote the platform free to buyers and it became far less expensive for suppliers, that we thought we were really cooking on gas (stupid English metaphor that means we thought we were really, really, good), sadly there were still a few gremlins waiting to leap out and bite us. Most of which to be fair to our own dev. team, were not of our own making.

Last year the UK Government “lost” 16,000 Covid records off an XL spreadsheet because someone forgot ( or maybe just no-one remembered) there was a limit to excel of 64,000 rows? The missing 64,000 were just too far down the list and fell off the bottom. Here’s a reminder in case you need one.

This could of course NEVER happen to us because we import using .CSV files which are in theory far more secure and don’t have a size limit. As you can see in the image above we publish a theoretical maximum file size of 64,000 rows based on what we thought our server could handle, but had never loaded any files flexing that barrier, until last week when this happened.

Ooops. How could that happen to us? Job #1 fix that bug on the live site. Job done.

So now for the next challenge, in theory ( good word “theory“) we could have no limit to the number of parts loaded on one upload file, but what would that do to the server, how would it be impacted in terms of performance, and what steps would we need to take to ensure that no malicious code, or fake listings were added without our knowledge? The second part of that question is not for public consumption, it must stay within the reach of our dev team and management team.

The first part is easier to promote. We are now testing files containing up to 1M parts, and are fine-tuning our server to ensure that processing these large files does not impact the other services that already run 24/7 ( part searches, public and private messaging, rfq’s and News). So far there is a cautious thumbs up. Even the dev team seem happy! This is a massive increase in parts upload capacity.

The blips that show up here are only those expected during the early Monday AM mailing run and the larger files we practised loading. The server coped well, and gave us the result below we were looking for

So what does any of this mean to our users? It means we can update larger files more rapidly. It means we can continue to expand the content and scope of our services, and most importantly it gives us the leg-up we need to be able to grow into more part-number-intensive areas of the supply chain, especially electronics where stock lists can easily run into the millions of line items.

Most important of all is the opportunity for trade buyers to use our service to source more parts more easily and for our suppliers it offers them a bigger, more global audience. If you are already listing with us, please send your latest stock update in NOW and if you are not yet listing then the time is right to start today.