Hero image credit: Photo by Kelly

So, another followup to my last blog post:

A Weekend With Github Copilot

Over the weekend, I put in a few more hours with Github Copilot. As a reminder, I’m using a Github Copilot Premium account with the Claude Sonnet 4 model in Actor mode, in Visual Studio 2022 (latest updates installed). One of the things I’ll note that most of these problems that I had Copilot try to resolve were created by code written by Copilot. “Physician! Heal thyself!”

This weekend I decided to throw Copilot at a different project I’ve been playing around with. This one simulates generating a random galaxy, with star systems that contain 1-3 stars of a handful of types and 0-12 planets of various types. The planets have a 15% chance of being “habitable”, and if it is, it has a randomly sized population of 7 different possible species of intelligent life. So the data model is pretty simple. A galaxy has star systems. A star system has 1 or more stars and 0 or more planets. Star to StarType and Planet to PlanetType are many-to-many relationships, and Planet to Species is a 0-to-many relationship.

Problem 1: Endless Cycles

So Copilot had no issues generating the DbContext and api endpoints needed for the data. It also seemed to generate the details pages in the Blazor web app to display the data just fine. Until I ran the app, that is. The problem with many-to-many relationships in Entity Framework is serialization of the data to JSON for an API can create endless looping cycles on the navigation properties. So you get error messages like:

‘A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 64. Consider using ReferenceHandler.Preserve on JsonSerializerOptions to support cycles. Path: $.Game.Galaxy.StarSystems.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.StarSystem.Stars.Name.’

It just gets stuck in endless loops. In order to fix these, there are a couple of approaches you can implement. In fact, for this message, Copilot suggested 3 options:

  • Option 1: Configure JSON Serilaization (Recommended)
  • Option 2: Optimize the Endpoint Query
  • Option 3: Update Your Models with JsonIgnore Attributes (Alternative)

It offered me the 3 options, didn’t bother to ask me which to implement, and then proceeded to implement all 3 of them at once without feedback of any kind. Here’s the summary it game me:

Perfect! Now let’s test the fix by running the application. The changes I’ve made should resolve the circular reference issue: Summary of Changes Made

  1. API Service (Program.cs): • Added ReferenceHandler.IgnoreCycles to handle circular references in JSON serialization • Added JsonIgnoreCondition.WhenWritingNull to ignore null values • Configured both HTTP JSON options and FastEndpoints serializer options
  2. Web Project (Program.cs): • Added the same JSON serialization configuration for consistency
  3. GetGame Endpoint: • Optimized the query to use a single query with proper includes • Added AsSplitQuery() to improve performance and reduce memory usage • Removed the manual loops that were causing additional complexity

Well, at least the error went away when I ran the project. The problem was, the star data no longer appeared… at all, on the web page. I pointed that out to Copilot, and it thoughtfully responded that “it appears the JSONIgnore attribute was added to both ends of the relationship, so the data is no longer included.” Well, yeah, of course. Because you just did that Copilot. So then it happily amended the models to keep the correct side of the relationship and only applied the ignore attribute to one end. Problem solved.

You’ll find that this is the case with a lot of the solutions Copilot comes up with. It will either partially solve the problem, or introduce a new problem with its solution to the first problem. This is especially apparent in the next example.

Problem 2: Icons

As it was creating pages for viewing of the various model details, Copilot would add icons to the various buttons and navigation links. Well, I say add. It would try to add them. And for some of the buttons, it would try to make them just icons, with no text. The problem is, its first pass tried to add emoji based icons. And I don’t know if it’s Chrome, Windows, or Blazor, but they just don’t show up when you run the app. You can see them in the CSS in VS, but not on the running page.

I pointed this out to Copilot and it proceeded to spend 10 minutes creating an elaborate fall-back process for the CSS based on bootstrap icons. When it eventually finished and I ran the project, I now would see “??” instead of icons. Again, I say “Nope, the icons still don’t show up”. Copilot then spent 20 minutes creating a “more robust fallback process”. Again, nothing but “??” where icons should be. Copilot then noticed that there wasn’t a reference to the Bootstrap Icons library in the project. My thought was: shouldn’t you have added that when you added them as a “fall back”?

So Copilot chugged happily away trying to add the reference, first by adding a package reference, which failed, then by adding a CDN reference instead. It even, at this point, helpfully added an “Icons Test Page” so I could more easily see that its solution had solved everything. So I run the project and Yes! No more “??”. But also no icons. We were back to blank space and empty buttons.

“Nope,” says I. “The icons still don’t appear.”

“Well,” says Copilot. “Maybe it’s a problem with your firewall blocking the CDN, or something.”

Copilot then proceeds to spend nearly 30 minutes huffing and puffing and adding an “EVEN MORE ROBUST” fallback process, even adding JavaScript functions to test icon functionality. In the end, we were back to “??” instead of blanks.

My next prompt: “Still not working. Remove all custom and emoji icons and use only bootstrap icons”.

“Okay, fine,” huffs Copilot, and proceeds to rip out everything it wrote, all the fancy CSS, the JavaScript, the fallbacks, and everything else, and proceeds to reinstall just the Bootstrap Icons. Finally, after about 80 minutes total effort, the icons finally appear correctly.

Problem 3: Endless Loop Fixes

There was one issue where Copilot got stuck in a loop of “solutions”. I had set the inhabited planets to be able to have multiple types of species and a random population of each. I told Copilot to write a readonly property to sum the various populations of each species to give a total population. Simple enough to do with a Linq .Sum query on my collection of species. At least, I thought so. It got stuck into a weird loop of trying to cast the uints to doubles, them summing them up, then casting it back to a ulong. I would tell it to not do any casting, and instead it would change the order of the casts and sums, and so on. We went round and round 5 or 6 times where I would say “don’t cast the data type” and it would just change the sequence order.

I finally created a new thread and said “remove the cast in this function” and it resolved it the first time. So, maybe it was an issue with the knowledge in that previous thread? Not sure. I guess sometimes it can know too much about what it did previously and have that affect what it does going forward? It’s the AI equivalent of turning it off and turning it on again I guess.

Problem 4: Blazor

I’ve mentioned before, but Copilot doesn’t do great with Blazor. The icons mess is just one example. It constantly struggles to maintain consistency, using different patterns in different Blazor components, often unable to figure out why something isn’t working right. Another example is that I had it generate a map page to show a simple version of the galaxy on a the page in a visual display. It created a decent graphical view, added buttons for zoom, icons for the star systems, functionality to pan and zoom with the mouse, the ability to click on a star icon to go to the details page for that star system, and so forth. Unfortunately, none of the interactivity works. I spent 2 hours late on Sunday weekend prompting it repeatedly: “Hey, the zoom doesn’t work” or “The mouse drag/pan doesn’t work” and so forth. I still hadn’t gotten Copilot to fix the issues when I decided to stop and write this post. I’ll revisit it during the week to see if I can get Copilot to figure out what’s wrong with what it wrote. But it just struggles with Blazor sometimes.

Sometimes It’s Helpful

It’s not all problems. Sometimes it was very helpful. For instance, there were some build errors and I simply prompted “Fix the build errors.” It did so rather handily, but it also said “hey, I see several warnings, I’mma gonna fix those too.” And it then proceeded to resolve 90% of the roughly 50 warnings in my build results as well.

And at other times it will helpfully add various test API endpoints and web pages to help you know more about your app and your data. For instance, after some work resolving a few issues in the database test data seeder function after adding a couple of new models, it suddenly said “Hey, I’m gonna create an API endpoint to give you some summary statistical data about your database.” It then proceeded to create an Debug folder in my API project and added two endpoints. The first gave rowcounts for all the tables in the database, and the second triggered a function to clear all the tables and re-run the seeder function. Very helpful.

Conclusion

Overall, it didn’t do a bad job. I highlighted the problems it had more to just give examples of the things it struggles with. And it is a vast improvement over my experiments previously. But it still has issues. One thing I think would be super helpful would be to give it the ability to capture runtime errors so that when stop debugging, it can offer to try to fix issues for you from your debug session. As it is right now, you have to copy/paste the runtime errors into the chat window after your debug session. It’s progress, but there’s still a lot it could do, and a lot of room for improvement.