Cooking, the Tandon way

Next time you're heating up leftovers, thank Ernst Weber.

Weber presided over what was then known as the Polytechnic Institute of Brooklyn, from 1957, when President Harry Rogers suddenly died and he was asked to step in, to 1969. It was on the day of Weber’s official inauguration as president of the Institute that Poly, as it was affectionately called, dedicated its new facility at 333 Jay Street.

The years of Weber’s presidency have been widely characterized as a time of unprecedented growth and intellectual achievement. Weber, as a New York Times reporter once wrote, helped turn the school “into one of the nation's foremost science-engineering centers” and “by the end of his tenure, the institute's enhanced status was nationally recognized.”

Born on September 6, 1901 in Vienna, Weber was a noted authority on microwave techniques, and his work on the calibration of radar systems was vital to U.S. efforts in World War II. He founded the school’s highly regarded Microwave Research Institute in 1945, and it ultimately grew into the country’s foremost centers of electronic research.

But of course, his research was used for something much more commonplace - and often taken for granted: the microwave oven. While not directly involved with it’s development, the fundamental research performed at the institute played a key role in how the technology would grow and spread.

Of course, the microwave isn’t the only item of convenience in your kitchen shaped by Tandon. John Gilbert (‘53) was asked by a Dupont representative to evaluate a new substance called Teflon. He seized the opportunity, studied the substance, and tested it on pots and pans. Gilbert approached Dupont with the invention of non-stick Teflon coated pots and pans, which has revolutionized cooking worldwide ever since.

As part of the NYU Tandon School of Engineering’s Vertically Integrated Projects (VIP) program, the Urban Food Lab is one-part vertical urban farm and one-part research laboratory. The Lab gives NYU students the chance to learn vertical farming techniques while designing their own research experiments that use the farm as a resource and contribute to its sustainability.

The Urban Food Lab is an aquaponic vertical farm comprised of fish tanks, hoses, grow lamps, and grow shelves. The fish feed on sustainably sourced fish food to produce nutrient-rich waste. The hoses then pump the waste out of the tanks. The waste travels through the hoses to the grow shelves where it is fed directly to the roots of the plants. The plants absorb the nutrient-rich waste, creating an incredibly efficient system that utilizes every last drop of its resources.

Research projects at the lab have included: advanced composting techniques, investigations into how the rhizosphere affects aquaponic environments, aeroponic systems development, optical fiber daylighting systems, and many more.

If you’ve ever learned to cook, you know how daunting even simple tasks can be at first. It’s a delicate dance of ingredients, movement, heat, and techniques that newcomers need endless practice to master.

But imagine if you had someone – or something – to assist you. Say, an AI assistant that could walk you through everything you need to know and do to ensure that nothing is missed in real-time, guiding you to a stress-free delicious dinner.

Claudio Silva, director of the Visualization Imaging and Data Analytics (VIDA) Center and professor of computer science and engineering and data science at the NYU Tandon School of Engineering and NYU Center for Data Science, is doing just that. He is leading an initiative to develop an AI “virtual assistant” providing just-in-time visual and audio feedback to help with task execution.

The NYU group – including investigators from NYU Tandon’s Department of Computer Science and Engineering, the NYU Center for Data Science (CDS) and the Music and Audio Research Laboratory (MARL) – have been performing fundamental research on knowledge transfer, perceptual grounding, perceptual attention and user modeling to create a dynamic intelligent agent that engages with the user, responding to not only circumstances but the user’s emotional state, location, surrounding conditions and more.

Dubbing it a “checklist on steroids” Silva says that the project aims to develop Transparent, Interpretable, and Multimodal Personal Assistant (TIM), a system that can “see” and “hear” what users see and hear through the use of a VR headset, interpret spatiotemporal contexts and provide feedback through speech, sound and graphics.

To hone TIM’s capabilities, Silva’s team trains it on a process that is at once mundane and highly dependent on the correct, step-by-step performance of discrete tasks: cooking. A critical element in this video-based training process is to “teach” the system to locate the starting and ending point — through interpretation of video frames — of each action in the demonstration process.

And while cooking may be a part of the project to provide proof-of-concept in a low-stakes environment, the work lays the foundation to one day be used for everything from guiding mechanics through complex repair jobs to combat medics performing life-saving surgeries on the battlefield.