A part of an automated tool chain needs some input data from a website accessible through a well-documented and working API to fetch data in "CSV" format. The output of the Invoke-WebRequest requires some further selection of the response data, since the response not only contains the desired "CSV" data but also some explanatory lines. Here's the data (exact URL to reproduce below):
Latitude (deg.): 51.654
Longitude (deg.): 6.606
A H_hor A_sun(w) H_sun(w) A_sun(s) H_sun(s)
-180.0 0.8 -180.0 0.0 -180.0 0.0
-172.5 0.8 -165.6 0.0 -172.9 0.0
-165.0 1.1 -152.1 0.0 -165.9 0.0
-157.5 1.1 -140.0 0.0 -158.9 0.0
-150.0 1.1 -129.4 0.0 -152.2 0.0
-142.5 1.5 -120.1 0.0 -145.7 0.0
-135.0 1.5 -112.0 0.0 -139.4 0.0
-127.5 1.5 -104.7 0.0 -133.3 0.0
-120.0 1.5 -98.1 0.0 -127.4 1.6
-112.5 1.5 -91.9 0.0 -121.6 5.4
-105.0 1.5 -86.1 0.0 -116.0 9.5
-97.5 1.5 -80.5 0.0 -110.5 13.8
-90.0 1.5 -74.9 0.0 -105.1 18.2
-82.5 1.5 -69.5 0.0 -99.5 22.7
-75.0 1.5 -64.0 0.0 -93.9 27.3
-67.5 1.1 -58.4 0.0 -88.1 32.0
-60.0 1.1 -52.6 0.0 -81.9 36.6
-52.5 1.1 -46.7 2.0 -75.3 41.2
-45.0 1.1 -40.6 5.2 -68.0 45.6
-37.5 1.1 -34.3 8.0 -59.9 49.8
-30.0 0.0 -27.8 10.4 -50.6 53.6
-22.5 0.4 -21.1 12.3 -40.0 56.9
-15.0 0.4 -14.1 13.7 -27.9 59.5
-7.5 0.4 -7.1 14.6 -14.4 61.2
0.0 0.0 0.0 14.9 0.0 61.8
7.5 0.0 7.1 14.6 14.4 61.2
15.0 0.0 14.1 13.7 27.9 59.5
22.5 0.0 21.1 12.3 40.0 56.9
30.0 0.4 27.8 10.4 50.6 53.6
37.5 0.0 34.3 8.0 59.9 49.8
45.0 0.0 40.6 5.2 68.0 45.6
52.5 0.0 46.7 2.0 75.3 41.2
60.0 0.0 52.6 0.0 81.9 36.6
67.5 0.0 58.4 0.0 88.1 32.0
75.0 0.0 64.0 0.0 93.9 27.3
82.5 0.4 69.5 0.0 99.5 22.7
90.0 0.4 74.9 0.0 105.1 18.2
97.5 0.0 80.5 0.0 110.5 13.8
105.0 0.8 86.1 0.0 116.0 9.5
112.5 0.8 91.9 0.0 121.6 5.4
120.0 0.8 98.1 0.0 127.4 1.6
127.5 0.8 104.7 0.0 133.3 0.0
135.0 0.4 112.0 0.0 139.4 0.0
142.5 0.4 120.1 0.0 145.7 0.0
150.0 0.8 129.4 0.0 152.2 0.0
157.5 0.8 140.0 0.0 158.9 0.0
165.0 0.8 152.1 0.0 165.9 0.0
172.5 0.8 165.6 0.0 172.9 0.0
180.0 0.8 180.0 0.0 180.0 0.0
A: Azimuth (0 = S, 90 = W, -90 = E) (degree)
H_hor: Horizon height (degree)
A_sun(w): Sun azimuth in the winter solstice (Dec 21) (0 = S, 90 = W, -90 = E) (degree)
H_sun(w): Sun height in the winter solstice (Dec 21) (degree)
A_sun(s): Sun azimuth in the summer solstice (June 21) (0 = S, 90 = W, -90 = E) (degree)
H_sun(s): Sun height in the summer solstice (June 21) (degree)
PVGIS (c) European Union, 2001-2021
I'd like to only select the lines matching '^[0-9-]' (positive and negative numbers). Essentially, selecting the following lines:
-180.0 0.8 -180.0 0.0 -180.0 0.0
-172.5 0.8 -165.6 0.0 -172.9 0.0
-165.0 1.1 -152.1 0.0 -165.9 0.0
-157.5 1.1 -140.0 0.0 -158.9 0.0
-150.0 1.1 -129.4 0.0 -152.2 0.0
-142.5 1.5 -120.1 0.0 -145.7 0.0
-135.0 1.5 -112.0 0.0 -139.4 0.0
-127.5 1.5 -104.7 0.0 -133.3 0.0
-120.0 1.5 -98.1 0.0 -127.4 1.6
-112.5 1.5 -91.9 0.0 -121.6 5.4
-105.0 1.5 -86.1 0.0 -116.0 9.5
-97.5 1.5 -80.5 0.0 -110.5 13.8
-90.0 1.5 -74.9 0.0 -105.1 18.2
-82.5 1.5 -69.5 0.0 -99.5 22.7
-75.0 1.5 -64.0 0.0 -93.9 27.3
-67.5 1.1 -58.4 0.0 -88.1 32.0
-60.0 1.1 -52.6 0.0 -81.9 36.6
-52.5 1.1 -46.7 2.0 -75.3 41.2
-45.0 1.1 -40.6 5.2 -68.0 45.6
-37.5 1.1 -34.3 8.0 -59.9 49.8
-30.0 0.0 -27.8 10.4 -50.6 53.6
-22.5 0.4 -21.1 12.3 -40.0 56.9
-15.0 0.4 -14.1 13.7 -27.9 59.5
-7.5 0.4 -7.1 14.6 -14.4 61.2
0.0 0.0 0.0 14.9 0.0 61.8
7.5 0.0 7.1 14.6 14.4 61.2
15.0 0.0 14.1 13.7 27.9 59.5
22.5 0.0 21.1 12.3 40.0 56.9
30.0 0.4 27.8 10.4 50.6 53.6
37.5 0.0 34.3 8.0 59.9 49.8
45.0 0.0 40.6 5.2 68.0 45.6
52.5 0.0 46.7 2.0 75.3 41.2
60.0 0.0 52.6 0.0 81.9 36.6
67.5 0.0 58.4 0.0 88.1 32.0
75.0 0.0 64.0 0.0 93.9 27.3
82.5 0.4 69.5 0.0 99.5 22.7
90.0 0.4 74.9 0.0 105.1 18.2
97.5 0.0 80.5 0.0 110.5 13.8
105.0 0.8 86.1 0.0 116.0 9.5
112.5 0.8 91.9 0.0 121.6 5.4
120.0 0.8 98.1 0.0 127.4 1.6
127.5 0.8 104.7 0.0 133.3 0.0
135.0 0.4 112.0 0.0 139.4 0.0
142.5 0.4 120.1 0.0 145.7 0.0
150.0 0.8 129.4 0.0 152.2 0.0
157.5 0.8 140.0 0.0 158.9 0.0
165.0 0.8 152.1 0.0 165.9 0.0
172.5 0.8 165.6 0.0 172.9 0.0
180.0 0.8 180.0 0.0 180.0 0.0
As an example, let's look at the following session explaining my confusion:
~#@❯ $horizonUrl = 'https://re.jrc.ec.europa.eu/api/printhorizon?lat=51.654&lon=6.606&browser=0&outputformat=csv'
~#@❯ $response = Invoke-WebRequest -Uri $horizonUrl
~#@❯ $respcont = $response.Content
~#@❯ $respcontstr = [String[]]$response.Content
~#@❯ $respcontobj = [Object[]]$response.Content
~#@❯ $respcont | Select-String -Pattern "^[0-9-]" -Raw
~#@❯ $respcontstr | Select-String -Pattern "^[0-9-]" -Raw
~#@❯ $respcontobj | Select-String -Pattern "^[0-9-]" -Raw
~#@❯ $respcont.GetType().Name " " $respcont.GetType().BaseType
String System.Object
~#@❯ $respcontstr.GetType().Name " " $respcontstr.GetType().BaseType
String[] array
~#@❯ $respcontobj.GetType().Name " " $respcontobj.GetType().BaseType
Object[] array
---> No output on the Select-String for the variants above <---
~#@❯ Invoke-WebRequest -Uri $horizonUrl -OutFile .\horizon.csv
~#@❯ $gc = (Get-Content .\horizon.csv)
~#@❯ $gc.GetType().Name " " $gc.GetType().BaseType
Object[] array
~#@❯ $gc | Select-String -Pattern "^[0-9-]" -Raw
-180.0 0.8 -180.0 0.0 -180.0 0.0
-172.5 0.8 -165.6 0.0 -172.9 0.0
-165.0 1.1 -152.1 0.0 -165.9 0.0
-157.5 1.1 -140.0 0.0 -158.9 0.0
-150.0 1.1 -129.4 0.0 -152.2 0.0
-142.5 1.5 -120.1 0.0 -145.7 0.0
-135.0 1.5 -112.0 0.0 -139.4 0.0
-127.5 1.5 -104.7 0.0 -133.3 0.0
[...]
---> Piped through Get-Content, Select-String dumps the desired output <---
How come Select-String works well together with an intermediate file in piped command chain, however not when used directly, without storing the response content of the Invoke-WebRequest?
The explicit type conversion above was my meager attempt at mimicking the data structure of the output of Get-Content. All the outputs before the pipe look identical in the console (lines of strings).
What did I miss?
CodePudding user response：
The Content property on the response object you get from Invoke-WebRequest is a single multi-line string. Get-Content on the other hand returns multiple single-line strings.
You can use the -split operator to split on newlines:
$respcont -split '\r?\n' |Select -Skip 4 |Select -SkipLast 9