One of the most basic – and banal - forms of news is the weather. A quick look at current conditions and a peek ahead at the forecast is something many people do repeatedly throughout the day. It’s one of the most common functions of voice assistants – “Okay Google, what’s the weather outside right now?” So how can we take a feed of weather information – in this case, an automated weather report from every airport in the country – and turn it into a conversational weather report that we can use on social or other places.
Platforms: ChatGPT 3.5, ChatGPT 4.0, Bard, and Llama-2
Data source: NOAA’s https://aviationweather.gov/
Goal: Casual weather report
The prompt
The challenge with this getting an LLM to use human language and common measurements. Given the information it has, it will use barometric pressure readings and metric units. The work here is in setting up the prompt. After trial and error, this is the prompt that ultimately worked for one LLM.
You are the weather reporter for a local news organization. Your job is to produce a three-sentence weather report using data from a NOAA Aviation Weather Report. You are not reporting weather for aviators.
Your report must obey the following rules.
- Do not report the airport name. Report the city the airport is in.
- Convert all metric measurements to imperial.
- The temperature readings given are in Celsius. Convert them to Fahrenheit.
- Convert all times and dates to Central US Time.
- Convert all wind speeds to miles per hour. Do not use knots.
- Do not report barometric pressure readings.
- Do not report altimeter settings.
- You do not need to report the state.
- You do not need to report the F to indicate Fahrenheit.
- The first sentence is about the current weather.
- The next two sentences are about the forecasted weather.
- Do not report the cloud ceilings.
- If three sentences of accurate weather information is not possible, write two.
- Only use information from the weather report to write your sentences.
Here is the weather information you have to use:
Current weather conditions: KLNK 131554Z VRB04KT 10SM OVC085 22/11 A3013 RMK AO2 SLP196 T02220106
Forecasted weather conditions: KLNK 131459Z 1315/1412 20005KT P6SM OVC080
Results
The basic facts of the weather report are that it’s about 72 degrees with variable winds at about 5 mph and overcast skies. The forecast is for more of the same, with winds out of the southwest at about 6 mph.
The results of testing this were varied. In spite of specific instructions to the contrary, ChatGPT 3.5 conflated Celsius and Fahrenheit, used terms it was told not to, and insisted that the forecast included temperature information when it did not. Other times, it invented a forecast of rain.
Examples:
In Lincoln, the current weather features overcast skies with a temperature of 22°F and a light variable wind at 4 mph. Over the next 24 hours, expect the conditions to remain mostly cloudy with a slight warming trend, reaching a high of around 80°F in the afternoon, and a steady wind from the south-southwest at 5 mph.
The temperature at the time was 22 degrees Celsius, not Fahrenheit. It should have been 72 degrees. The wind report was correct. As was the wind report in the forecast. However, nowhere in the forecast is there information about a warming trend.
Currently, in Lincoln, the temperature is 22°F with overcast skies, and the wind is variable at 4 mph. For the forecast, expect overcast conditions with a high of 22°F and a low of 11°F, while the wind will continue to be light, around 5 mph, throughout the day.
With another edit to the prompt, I added text that specifically listed the current weather conditions were the first line and the second line contained the forecast conditions. That addition improved the prompt considerably, with one major problem.
Currently in Lincoln, the weather is overcast with a temperature of 72°F and a light variable wind at 4 mph. For the forecast, expect overcast conditions to continue with a high of 80°F and a light southerly wind at 5 mph through the day, and the possibility of some showers overnight.
The problem: Nowhere in the forecast is any mention of rain.
Other models
Bard, as it is wont to do, turned the report in to bullet points.
Sure, here is the three-sentence weather report for Lincoln, Nebraska, as of 15:27 CDT on September 13, 2023:
The current weather in Lincoln is mostly cloudy with a temperature of 72 degrees Fahrenheit and winds of 4 miles per hour.
The sky will remain mostly cloudy throughout the afternoon, with winds increasing to 10 miles per hour.
The temperature will drop to 60 degrees Fahrenheit overnight.
Here is the two-sentence weather report:
Mostly cloudy skies with a temperature of 72 degrees Fahrenheit and winds of 4 miles per hour.
Winds will increase to 10 miles per hour in the afternoon.
I hope this is helpful!
There is nothing in the forecast about winds increasing to 10 mph during the afternoon, nor anything about a temperature drop overnight.
Llama-2-7b via gpt4all was a complete mess.
Sure, I’d be happy to help! Here is the three-sentence weather report based on the provided data:
Currently (at 1315 UTC), the airport in Kansas City is experiencing mostly cloudy skies with a temperature of 22°F (-6°C) and a wind speed of 4 knots. The barometric pressure is 30.13 inches of mercury, and the altimeter setting is 29.97 inches.
For tomorrow (at 1315 UTC), there is a forecasted chance of light snow showers with a high temperature of 34°F (1°C) and a low temperature of 20°F (-7°C). The wind speed will be around 6 knots, and the barometric pressure will remain steady at 30.13 inches of mercury.
I hope this helps! Let me know if you have any other questions.
Llama-2 clearly has some training data that parses those weather reports. But it ignored all the rules given to it and Lincoln is very much not Kansas City.
Further testing
To test the prompt, I used weather reports from other airports to test it. For example, this is the report from Chicago’s Midway Airport.
In Chicago (Midway Airport), it’s currently partly cloudy with a temperature of 70°F and a north-northeast wind at 8 mph. The forecast calls for mostly clear skies through the day with a high around 68°F and a light north-northeast wind at 4 mph. Expect some cloud cover to increase in the late afternoon and evening, with a chance of light rain showers overnight as the wind shifts to the northeast at 6 mph.
The bulk of the report is correct, but yet again, the forecast doesn’t call for rain in spite of this report.
A working version
ChatGPT 4.0 is the only LLM tried that worked. It obeyed the rules, converted the units correctly and didn’t invent any weather not in the report.
The current weather in Lincoln is mostly cloudy with a temperature of 72°F and light variable winds at around 5 mph. The forecast for today shows overcast skies with winds coming from the south at about 6 mph. This weather condition is expected to last until tomorrow morning.
A more difficult example
This weather report from Boston is more complex than the previous two, in that it actually includes rain.
Current weather: KBOS 131810Z 21010KT 3SM RA BR FEW009 SCT026 OVC080 23/20 A2993 RMK AO2 VIS E-SE 2 P0008 T02280200`
Forecast: KBOS 131733Z 1318/1424 14004KT P6SM -SHRA BKN070 BKN100 OVC140 TEMPO 1318/1320 2SM TSRA BR BKN020CB OVC060 FM132200 18006KT 5SM -SHRA BR VCTS OVC060CB FM140000 18006KT 5SM -SHRA BR OVC070 FM140400 25004KT 5SM BR OVC035 FM140800 29005KT 6SM BR BKN100
At the time of this report, it was raining and misting in Boston. The forecast called for more light rain, thunderstorms and light showers giving over to mist later with the wind changing directions. Here’s how ChatGPT 4.0 handled it.
The current weather in Boston is rainy with a visibility of 3 miles, temperatures at 72°F, and winds from the southwest at 11 mph. Today’s forecast indicates light rain showers with occasional thunderstorms and reduced visibility down to 2 miles, coupled with an overcast sky. This evening, expect light rain showers, misty conditions, and overcast skies that will persist into tomorrow morning, with winds gradually shifting from the south to the west.
Spot on.
Conclusion
It’s clear ChatGPT 3.5, Bard and Llama-2 all have some understanding of aviation weather reports and is able to parse some of the data. It is also clear that it has limited understanding of what is Celsius and what is Fahrenheit and will conflate the two. Several of them struggle with adding weather that is not there. At this time, it appears ChatGPT 4.0 is the only LLM that does this as envisioned here.