I think your response was perfect; you shared, while making your own needs and expectations known. Now that you know that the workers would like to take the fruit, it would be good to state up-front how much, if any, you are willing to share with them. And you're probably right that the worker is low-income. It's possible he's struggling to support a family, whether he's here legally or not. He's still a human being, and so are his children. He didn't ask for his station in life, or else he'd have been born rich.
We live in a lush country area, and have neighbors on two sides of us who have fruit they have welcomed us to share. We also have trees of our own, and happily share the surplus when we have productive years.
There are also many trees along our walking route with fruit that goes to the birds, squirrels and deer, or to rot. If it's clear to us that nobody is using the fruit from an unfenced tree, we will take some and leave some to the wildlife. There are bazillions of volunteer cherry and plum trees around here, and a few decent wild apple trees. We do live on a very low income, so this is one way to enrich our diets.
But if there is any question about whether a roadside tree is valued by the property owner, if it's on maintained property or behind a fence, we'll leave it alone. Even though it breaks our hearts to watch the fruit go to waste.
I'll bet that yard worker is terrified that he'll lose his job. And it may be routine in his line of work for low-paid employees to take fruit that they assume won't be used.