Skip to content
Command line/Python...
 
Notifications
Clear all

Command line/Python (Linux) GDAL Warp result much bigger than in QGIS (Windows)

1 Posts
1 Users
0 Likes
0 Views
Guest
Illustrious Member
Joined: 4 months ago
Posts: 57426
Topic starter  

I'm trying to run the following code to downsample a ~3GB GTiff while coregistering it to a reference extent:

def coreg_downsamp(source, destination, output_type="GTiff"):
    #essentially we're Warping into the same CRS, but with a resolution of 10m,
    # and with destination extent defined by the ELME reference raster
    # (also, using multi-threaded processing and finally compression)
    
    #attempt with python bindings
    #opts =  f"-tr 10.0 10.0 -r med -te {REFERENCE_EXTENT_STR} -dstnodata -9999.0 -multi -of {output_type} -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9"
    #gdal.Warp(destNameOrDestDS = destination, srcDSOrSrcDSTab = source, options=opts)
    
    #just sending it straight to command line
    command = f"gdalwarp -dstnodata -9999.0 -tr 10.0 10.0 -r med -te {REFERENCE_EXTENT_STR} -multi -of {output_type} -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 {source} {destination}"
    subprocess.call(command, shell=True)

I built the command using the QGIS Warp tool, which you can see below:

enter image description here

Running the QGIS GDAL Warp tool works great! It does everything I need, and the result is even about ~2GB after compression. However, if I take the exact same command and run it in the command line, send it to the command line from Python, or use the Python GDAL bindings, the result is more like ~10GB and I have to turn on the BIGTIFF flag for it to even work:
enter image description here

The only difference at all is that I'm running my command line/Python in Linux (WSL2 Ubuntu) whereas my QGIS is running on Windows, so I've changed the I/O filepaths accordingly. Both are using GDAL 3.4.3.

For good measure, here's the GDALInfo on the input file:

Driver: GTiff/GeoTIFF
Files: eesti_ndvi_suvi2022.tif
Size is 37013, 24284
Coordinate System is:
PROJCRS["Estonian Coordinate System of 1997",
    BASEGEOGCRS["EST97",
        DATUM["Estonia 1997",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4180]],
    CONVERSION["Estonian National Grid",
        METHOD["Lambert Conic Conformal (2SP)",
            ID["EPSG",9802]],
        PARAMETER["Latitude of false origin",57.5175539305556,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",24,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",59.3333333333333,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",58,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",6375000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["northing (X)",north,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["easting (Y)",east,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Topographic mapping (large scale)."],
        AREA["Estonia - onshore and offshore."],
        BBOX[57.52,20.37,60,28.2]],
    ID["EPSG",3301]]
Data axis to CRS axis mapping: 2,1
Origin = (369010.000000000000000,6620000.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  369010.000, 6620000.000) ( 21d40'24.68"E, 59d41'48.08"N)
Lower Left  (  369010.000, 6377160.000) ( 21d48'50.50"E, 57d31' 4.02"N)
Upper Right (  739140.000, 6620000.000) ( 28d14'36.00"E, 59d38'57.00"N)
Lower Right (  739140.000, 6377160.000) ( 27d59'15.00"E, 57d28'23.20"N)
Center      (  554075.000, 6498580.000) ( 24d55'50.87"E, 58d37'25.43"N)
Band 1 Block=37013x1 Type=Float32, ColorInterp=Gray

And on the (aborted) output file:

Driver: GTiff/GeoTIFF
Files: ndvitest_commandline.tif
Size is 37100, 25800
Coordinate System is:
PROJCRS["Estonian Coordinate System of 1997",
    BASEGEOGCRS["EST97",
        DATUM["Estonia 1997",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4180]],
    CONVERSION["Estonian National Grid",
        METHOD["Lambert Conic Conformal (2SP)",
            ID["EPSG",9802]],
        PARAMETER["Latitude of false origin",57.5175539305556,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",24,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",59.3333333333333,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",58,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",6375000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["northing (X)",north,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["easting (Y)",east,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Topographic mapping (large scale)."],
        AREA["Estonia - onshore and offshore."],
        BBOX[57.52,20.37,60,28.2]],
    ID["EPSG",3301]]
Data axis to CRS axis mapping: 2,1
Origin = (369000.000000000000000,6635000.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  PREDICTOR=2
Corner Coordinates:
Upper Left  (  369000.000, 6635000.000) ( 21d39'50.66"E, 59d49'52.42"N)
Lower Left  (  369000.000, 6377000.000) ( 21d48'50.21"E, 57d30'58.84"N)
Upper Right (  740000.000, 6635000.000) ( 28d16'31.78"E, 59d46'58.91"N)
Lower Right (  740000.000, 6377000.000) ( 28d 0' 5.93"E, 57d28'16.39"N)
Center      (  554500.000, 6506000.000) ( 24d56'23.65"E, 58d41'25.04"N)
Band 1 Block=37100x1 Type=Float32, ColorInterp=Gray
  NoData Value=-9999

Anyone have any clue what's going on here?


   
Quote

Unreplied Posts

Let $F$ be a field; what does $(F)$ mean?

$begingroup$

Let $L,K,L_1,L_2$ be fields where $L/K$ is a field extension and $K leq L_1,L_2 leq L$ be intermediate fields.
We define $L_1L_2$ to be the subfield of $L$ generated by $L_1 cup L_2$

Show the following:
$L_1L_2=L_1(L_2)=L_2(L_1)$

My thoughts:
I am a little bit confused what the expression $L_1(L_2)$ means. I have not seen $()$ being defined for fields, nor what the multiplication of fields is.
The thing that comes to mind is:
Let R be a ring. For $S subset R$

$(S):=bigcap {A: A$ is an Ideal of $R$ with $S subseteq A}$

In this case does $L_1L_2$ mean the multiplication of ideals which would be $L_1L_2:={a_1b_1+…+a_nb_n:a in L_1, b in L_2}$

Is the meaning of the notation the same for fields as for rings?

$endgroup$

Proving my function is optimized in a particular regime

$begingroup$

Consider the optimization problem:
$$
max_{x geq 0} (1-x)fleft(gleft(xright)right)
$$

where $f'(x)< 0, f(1) = 0, f(0) = 1$, and $f$ is differentiable. In addition, $g$ has the form:

$$
g(x)=
begin{cases}
1 & text{if } x < frac{c}{h_{max}}\
h^{-1}(c/x) & text{if } x in [frac{c}{h_{max}},frac{c}{h_{min}} ]\
0 & text{if } x > frac{c}{h_{min}}
end{cases}
$$

where $h^{-1}$ is the inverse function of some function $h$ such that $h(x) > 0$, $h'(x)>0$ and $h$ is differentiable. Assume also that $g$ is continuous.

Now if $c > {h_{max}}$, the function is zero or negative over the entire domain, so the solution is characterized.

I am interested in the case where $c$ is small. It seems to be that for “small” $c$, the solution should be $x = c/h_{min}$.

A condition for this to be true is
$$
frac{1 – frac{c}{h_{min}}}{1 – x} > f(g(x));; forall x < frac{c}{h_{min}}
$$

I wonder if I can say for any fixed, $f, g, h$ under these assumptions then there exists some value $k$ such that $forall c<k$, $x^* = c/h_{min}$.

$endgroup$

Unable to access an instanced health system

$begingroup$

I am having an issue with a health system I am making for my game. I have been at it for 3 days now and have not made much progress. For reference, I am following this tutorial by code monkey: Code Monkey Health Tutorial. The goal is to have two buttons, one for healing and one to damage the player, and a Text Mesh Pro UI Text to show the health of the player, just for testing purposes. Unlike the tutorial, the buttons and text are just on a canvas, not attached to the player, as I am setting it up as a form of a HUD system. also, forgive the formatting, during troubleshooting, I was more focused on trying to fix my issue and my code became a bit of a mess. Sorry for that.

I have three scripts

    using UnityEngine;


    public class HealthSystem
    {
        private int _health;
        private int _healthMax;

        public HealthSystem(int healthMax)
        {
            this._healthMax = healthMax;
            _health = healthMax;
            Debug.Log("healthSystem Created");
        }
        public int GetHealth()
        {
            return _health;
        }
        public void Damage(int damageAmount)
        {
            _health -= damageAmount;
            if (_health < 0) _health = 0;
        }
        public void Heal(int healAmount)
        {
            _health += healAmount;
            if (_health > _healthMax) _health = _healthMax;
        }
    }

    using UnityEngine;

    public class GameHandler : MonoBehaviour
    {
        private HealthButtons _healthButtons;

        private void Start()
        {
            HealthSystem _healthSystem = new HealthSystem(100);
            Debug.Log("Health = "+_healthSystem.GetHealth());
            _healthButtons.Setup(_healthSystem);
        }
    }
}
using UnityEngine;
using TMPro;


    public class HealthButtons : MonoBehaviour
    {
        [SerializeField]
        private TextMeshProUGUI numberText;
        private static HealthSystem _healthSystem;
        private int intToStringHealth;


        public void DamageButton()
        {
            _healthSystem.Damage(10);
            Debug.Log(_healthSystem.GetHealth());
        }
        public void HealButton()
        {
            _healthSystem.Heal(10);
            Debug.Log(_healthSystem.GetHealth());
        }
        public void Setup(HealthSystem healthSystem)
        {
            if (healthSystem == null) Debug.Log("healthSystem");
            Debug.Log(healthSystem);
            _healthSystem = healthSystem;
            Debug.Log("Setup is running");
        }
        private void Update()
        {
            intToStringHealth = _healthSystem.GetHealth();
            numberText.text = intToStringHealth.ToString();
        }
    }

my issue currently is Im trying to create an instance of HealthSystem in GameHandler, which I am successful in, then pass the information to HealthButtons using _healthButtons.Setup(). This is where I become unsuccessful as I am getting a NullReferenceException error.

from my efforts of troubleshooting currently, I have determined I am not calling the instance correctly, but I do not want to use code monkey’s libraries. I have tried making all classes monoclasses and attaching them to game objects and using SerializeFields for all my variables, which does work, but I am not sure that is best practice as I do know linking everything up in the inspector can be a chore if something goes wrong.

This tells me the solution should be fairly simple, I am just not able to articulate the question into google to get the correct answer. It may be a bit overkill just to test a function, but I would like to understand what I am doing wrong and get better. Thank you very much for your help.

$endgroup$

Deriving the log likelihood of the observed data

$begingroup$

enter image description here

Hi, I am deriving the log likelihood of the observed data in part a. However, I am not sure if I am deriving it the right way.

Below is my solution:

Yi ~ N(mu, sigma^2) = f(Yi = xi) = ∅ (xi; mu, sigma^2) (for ri =1)

Yi ~ N(mu, sigma^2) = f(Yi = xi) = Φ (xi; mu, sigma^2) (for ri =0)

Then I multiply these two terms

f(xi,ri) = ∅ (xi; mu, sigma^2)^ri*Φ (xi; mu, sigma^2)^1-ri

and then I form the likelihood equation:

L(mu,sigma^2|x,r) = n i=1 prod ∅ (xi; mu, sigma^2)^ri*Φ (xi; mu, sigma^2)^1-ri

Is this the correct way to derive the equation in part a?
I have also attached the picture of the question.

$endgroup$

How would you authenticate requests for player specific data in a server authoritative game?

$begingroup$

I am creating a server authoritative game that uses a public game service API to manage player specific data in the game. The client will authenticate with the auth server, get a token, then use that token to access player specific data in the API. This works well if the API calls to the game services occur on the game client. However, in a server authoritative model, the game server itself would need to make those calls on behalf of the client. My current idea is to replicate the access token to the server and then have the server figure out which token to use when it is requesting data for a specific client. I was curious if this is the best way to do this or is there another approach I should be considering?

$endgroup$

TypeError: ‘NoneType’ object is not iterable using Zonal Statistics

I’m trying do looping zonal statistics to many polygons in a folder, and rasters in aonther folder. Each polygon from a specific folder need to get the mean values from a many rasters from another folders, like slope, standard deviation, etc. But when I run this code the error is:

Traceback (most recent call last):
File “D:/ZONAL1.py”, line 10, in
for shp in shapefiles:
TypeError: ‘NoneType’ object is not iterable

What this means?
My code:

import arcpy
import os

arcpy.env.workspace = r'D:pontos_modelagembuffers'
arcpy.env.workspace = r'D:slopes'
arcpy.env.overwriteOutput = True
output = "D:tractebelpontos_modelagembufferdbf"
rasters = arcpy.ListRasters()
shapefiles = arcpy.ListFeatureClasses()  
for shp in shapefiles:
    for rst in rasters:  
        table_out = output + "\" + shp + "_Zonal_Est"   
        arcpy.gp.ZonalStatisticsAsTable_sa(shp,'FID',rst, table_out,"MEAN") 

Which is the real Euler’s Rotation theorem?

$begingroup$

There are two quite different, but useful, theorems related to 3D rotation that are referred to as “Euler’s Rotation Theorem“.

The first definition is given as a quotation in the book by Kuipers

  • Any two independent orthonormal coordinate frames can be related by a sequence of rotations (not more than three) about coordinate axes, where no two successive rotations may be about the same axis.

There is a similar, but more pithy definition, from Wolfram MathWorld

  • An arbitrary rotation may be described by only three parameters.

This is related to the three-angle representation of an arbitrary rotation by Euler, roll-pitch-yaw, Tati-Bryan angles etc.

The second definition is from Wikipedia

  • When a sphere is moved around its centre it is always possible to find a diameter whose direction in the displaced position is the same as in the initial position. (Also given in latin)

This related to the angle-axis representation of an arbitrary rotation. This version of the Rotation Theorem is found in many other places as well, including here on Math Overflow.

Both definitions are related to rotation, and the second at least has a reference to Euler 1776. The first is related to Euler axes and Euler angles, seems like the sort of thing Euler might have figured out, but Kuipers does not provide a reference.

Euler was notorious for not publishing much. Somebody, post Euler, has declared these theorems as Euler’s Rotation Theorem. Which one is it? Is there a more nuanced way to refer to them?

$endgroup$

Share: