Sorting 2 Tons of Lego, Many Questions, Results

darwhy | 322 points

I'm begging you, please post a full-speed video that shows a seemingly random assortment of pieces sorted into boxes with identical color/size/shape pieces. I think it would be insanely satisfying for many people to watch. I would literally grab popcorn and watch it for hours (and I think many others would too).

Given your current setup, it might be easiest to just take one bin of identical pieces and then sort it by color into various bins. I'm salivating over what this would look like.

If you posted that video, I bet it would rack up millions of YouTube views.

tuna-piano | 7 years ago

> The next morning I woke up to a rather large number of emails congratulating me on having won almost every bid (lesson 1: if you win almost all bids you are bidding too high)

That reminds me of the economist George Stigler's observation: "If you never miss a plane, you're spending too much time at the airport".

tzs | 7 years ago

A few years ago a friend and I started work on a similar machine for sorting Magic: the Gathering cards. We weren't using any machine learning (this was beyond our capabilities) but our design was similar, except the bins were on a moving platform which slowed down as the weight of the cards in bins increased - due to the cheap parts/weakness of the motors we were using. Unfortunately it occupied an entire room in my house and was never completed and ultimately dismantled.

cfeduke | 7 years ago

If, like me, you find photography of piles of sorted LEGO pieces hypnotically soothing, you will probably like this album I put together:

http://imgur.com/a/OJaq6

A while back, I sorted my childhood LEGO pile before sending it to my brother. 73 pounds of LEGO sorted by hand. I'm sure it says things about how strangely wired my brain is, but this was an eminently enjoyable endeavor.

munificent | 7 years ago

This series is the best thing ever.

An idea: I have a three-month-old daughter who I'll definitely be buying Lego for when she gets old enough. What I'll be looking for is whatever's the least valuable in your collection -- big bins of cheap mixed stuff to see what she invents. Maybe that would be a cool side output for your project? Cheap chaff packs for kids of hackers ...

JackC | 7 years ago

Not a fanatic builder at all, been almost two decades since I built something with Lego, but really surprised to see that the categories are not color separated.

If you're building any esthetic Lego project would have color restrictions. Also I imagine some colors are worth more than others, just because they're used in more projects.

Also, color separated Lego bins just look pretty.

tinco | 7 years ago

Why not go big? Write up a business plan that involves buying 100 tons of lego, making a bigger machine with 200 bins off the belt, or whatever, and making a new brickowl / bricklink site of your own. I'm sure you could find investors, you've got great publicity already. And you could propose it as a general sorting AI (or web sale of small item) firm in the long term, with the lego being your initial application.

You'd need to solve the inverse problem of how to efficiently go from bin to packing container, which doesn't seem too hard. Rotating carousels of bins with little robots or something. Set the item on a tray, and if isn't the right item, puff it back into the bin.

marze | 7 years ago

Are there CAD models available for the various Lego components? I'm wondering if, given such models, you could use rendering software to generate labeled training images. That could allow you to get a large training set with many images for each kind of component in many different orientations, even for components you do not yet have.

If the CAD models are not available, how much work is it to make a CAD model given an instance of a particular component? I'd expect that for some components, such as those NxM plates, you'd only need to build one model by hand in CAD software, and then could algorithmically generate the models for other sizes.

tzs | 7 years ago

Jacques, John Tooker might be able to help you with the sorting issue: http://lego.jtooker.com/sorting/.

BillFranklin | 7 years ago

I actually just gave away all my Lego to a colleague whose three sons are much better able to appreciate it than I am, and (because!) it's been quite a few years since I actually built anything.

But, that said, were I looking to buy parts, I'd hesitate to do so in bulk by these categories, unless the price were very good indeed - between the inability to match a parts list to a purchase order, and the extensive manual labor required to sort parts sold by mass at a useful degree of granularity, they'd need to be very cheap in order to make the purchase worth my while.

(On the other hand, most of my builds back in the day were spacecraft of various fictional types, and I heavily favored complexity, which means a large number of small parts, designing in wiring channels for LED lighting, and the like. I'm not sure how representative my comments here are of any other style.)

throwanem | 7 years ago

At some point you can completely reverse entropy and re-create the original sets :-) But in terms of just selling off excess Lego looking at the mix of the builder sets would let you figure out a sort of SKU you could put together and sell on Ebay or else where as a 'builder pack 1', 'builder pack 2' etc.

ChuckMcM | 7 years ago

One idea for selling the parts would be to keep a precise inventory of everything entered into the system. A precise inventory + a database of the parts needed for any LEGO set would allow customers to order an arbitrary set for a nice price. A total system which would take random parts as input, and output packaged complete LEGO sets would be neat.

pveierland | 7 years ago

On commercial pecan farms, the nuts need graded based on color, size, and variety. Currently, pecans are tumbled in rotating chutes composed of grates of increasing spacing, so that pecans are sorted based on size. Color and variety are not filtered, ultimately resulting in a lower sales price for the farmer. Your device could take this type of sorting to the next level.

jhassell | 7 years ago

With the machine working so well, you could run each sorted batch through it again, and sort into very specific pieces/colours, if you wanted.

You just need to re-train on those classes instead- but I bet the front-end of the tensorflow network would already be trained well enough that re-training for new classes would be very fast.

mabbo | 7 years ago

I've thought about this machine a number of times since the previous post. In that post, there was a movie showing the machine's operation, but it was operating at a slow speed, and he's mentioned multiple times that there's been speed improvements.

Does anyone know if there's a realtime movie anywhere?

nappy-doo | 7 years ago

My five year daughter old mostly just wants to build castles, so if there was a less-expensive way to get large number of 2-peg width bricks (especially in pink and purple, which she reminds me daily are her favorite colors), I could see that being useful.

jkingsbery | 7 years ago

> Overall the machine works well but I’m still not happy with the hopper and the first stage belt. It is running much slower than the speed at which the parts recognizer works (30 parts / second) and I really would like to come up with something better.

It sounds like your existing hopper is reliable but slow? If that's the case, could you just make two (perhaps somewhat smaller) hoppers and merge their outputs?

justinpombrio | 7 years ago

I suspect Mobile Frame Zero ( http://mobileframezero.com/mfz/ ) players would be delighted to get a source of the more esoteric bricks that go into their builds

Marazan | 7 years ago

I wonder if Jaques tried to do dropout to increase the efficiency? Also if he made the trained neural net available you could made a lego identifier app.

zitterbewegung | 7 years ago

In part 1 you mention that under vibration the bricks self assemble into bridges that obstruct the conveyor belt, and you want to minimize that.

How about maximizing it instead? Try to build bridges as long as possible? Or some other fitness function?

In theory you could control the vibration sequence and the part feeding sequence (once you have sorted all the parts :) ) and use an evolutionary algorithm...

Gravityloss | 7 years ago

Something I've wanted to try for years is what about a Lego refraction distillation tower? Similar to how oil refineries use a tower to sort hydrocarbon compounds by length based on their volatility. Except in this case instead of using heat we would use air blowing upward in a long tube to make the pieces "volatile". Picture a 6" diameter clear tube stretching floor to ceiling with a powerful blower at the base. Or a smaller version could be prototyped using an air popcorn popper with the heating element disabled.

Essentially it would sort pieces according to their free-fall terminal velocity. You could exploit the Venturi effect to make the bottom of the tube have a higher velocity than the top, or simply add bleed-air holes at progressive levels. There could be take-off doors at different levels of the tube. A nice property is that if the ingress and egress mechanisms are designed elegantly, it could run continuously.

(Although, that last point is one reason I've never built it: it seems like there would be a fine line between "air powered sorter" and "air powered rock tumbler" and I worry it could wear the corners of all my Lego bricks if it goes wrong.)

Another potential issue is that there might not be much logical relationship between the pieces which share similar terminal velocities, and different pieces may have different or metastable terminal velocities depending on initial orientation.

Obviously this wouldn't be a perfect sorter, but it could be a good pre- or post- sorter for your visual system. And for my purposes it might work well since instead of adapting the system to suit my sorting preferences I could simply train my sorting expectations based on what I learn the machine tends to sort together.

Hmmm now that I have an industrial dust collector installed in my workshop, I actually already have the bulkiest / most expensive part of this invention. I just need a tall acrylic tube and I could try this!

phaedrus | 7 years ago

> Judging by the increase in accuracy from 16K images to 60K images there is something of a diminishing rate of return here.

This is a well-known thing in machine learning. As far as I know all ML algorithms have this property, where the learning curve is basically logarithmic and each additive increase in performance requires multiplicatively more training material.

arnsholt | 7 years ago

Pretty cool to keep seeing follow-ups on this.

Jacques, any plans to publish your models or image dataset somewhere?

unityByFreedom | 7 years ago

I have a suggestion but am not a Lego guy. Maybe you can do a few things now that the sorter works, potentially using it to automate or minimize order fulfillment.

The idea is that if you can get a dataset of Lego packs / models you can use your inventory to make them. You can offer sales of the kits based on dynamic info. You can also provide inventory dynamically to users to purchase parts or upload a build manifest which will be filled.

I think the 2 huge assets you have are sorting for processing AND fulfillment, and real-time inventory levels of verified authentic parts.

Also, I would make some sort of contest, auction fun game where a person can win the whole pile of discarded stuff. Idk what's in there but I bet someone would want it.

wand3r | 7 years ago

I would pay good money to send you a big box of Lego from mixed sets and have it come back sorted into one bag per set.

You'd probably need the set numbers so that it'd be easier to choose which set to associate a part with.

I'm more than happy to be a beta customer!

sanj | 7 years ago
[deleted]
| 7 years ago

You can really earn money by modifying this machine to sort various nuts and fruits by quality in the food industry...

pulse7 | 7 years ago

On speeding up your feed, how about adding another belt prior to the main belt, with a variable speed?

You drop parts onto the variable speed belt from enough sources to get a good feed rate, and constantly vary the speed of that belt with video feedback to drop them onto the main belt with reasonably even spacing.

marze | 7 years ago

Now you have a bunch of pieces each with different classifications.

Possibly you could feed in parts lists from a huge amount of existing sets - have some sort of (handwavey learning thing or even markov chain), and generate mixes of pieces based on data sets from spaceships / cars / etc.

stuaxo | 7 years ago

Why not sort Lego the same way we sort plasic in Norway? Watch this video. Drawing of the technique at 0:35 and shown from 0:52 https://youtu.be/Mi0FHNLkim0

punnerud | 7 years ago

I think the sorted photos are great art. Maybe more valuable than the contents ;)

codewithcheese | 7 years ago

Technics and larger wheels are useful for Lego robotics (FLL) teams everywhere; certainly useful for our team.

Edit: this is in response to your asking for feedback about categories/demand.

Poiesis | 7 years ago

You should extend it so it can assemble the pieces ;)

mherrmann | 7 years ago

Has Lego approached you at all? If they don't have similar in-house technology, I could see this adding value to them.

tuna-piano | 7 years ago

Impressive, but I'm still wondering what 2 tons of lego look like.

jokoon | 7 years ago

You should use https://www.cloudflare.com/. It's free and very easy to implement.

andresgottlieb | 7 years ago