I would think that field work would be the most difficult part- but seeing as though I haven't obviously had the experience yet - I'm not sure what to say....

From my own and my cohorts experience this is what I have seen and found myself:
For many the most difficult part is lack of experience in the field. Often students become overwhelmed when they find that that "social work" is not what they expected or wanted. Another large issue is that of your own health and well-being. You will be forced to deal with your own issues if you have not already which is often difficult. i.e., you might be forced to intern at a place or with a population that "pushes all your buttons". And another large part is the workload which can be quite strenuous depending on whatever else you have going on in your life. But in spite of the difficulty, most students find it fun, exciting, and challenging. The best thing you could do is to go and meet social workers and try to really learn what they do. Talk to as many people as you can to get a balanced pictures. Volunteer if possible, and very importantly ask yourself, and examine why it is that you want to become a social worker. Hope this helps.

Thanks a bunch! It really does help to hear from others in the field. Hopefully, I will get a chance in the next few weeks, to speak to a few in person.