Evaluating global expressions is broken when the where string does not filter out all missing values

[jmorrisnrel]

What happened?

There seem to be issues currently with using NOT or OR/AND in the top level global_expressions where string related to the rework of _from_pyomo_expr() in pyomo_backend_model.py made in #551 . The expression below worked prior to #551 :

  # This expression worked before the rework of _from_pyomo_expr()
  # with a NOT in the top level expression where clause
  flow_out_non_transmission_techs:
    equations:
    - expression: flow_out
    foreach:
    - nodes
    - techs
    - timesteps
    - carriers
    where: NOT base_tech=transmission

but now causes a TypeError: 'int' object is not callable in _from_pyomo_expr()

The issue seems to be specifically with NOT/OR/AND as doing a straight base_tech filter works. We can also work around it by moving the NOT logic to an equations where string but I wanted to flag the issue with the top level where filtering.

Other examples of expressions:

  # A different approach to the expression that also fails with the OR
  # in the top level where clause
  flow_out_non_transmission_techs:
    equations:
    - expression: flow_out
    foreach:
    - nodes
    - techs
    - timesteps
    - carriers
    where: base_tech=supply OR base_tech=storage OR base_tech=conversion

  # An expression that still works without any NOTs or ORs
  # in the top level expression where clause
  flow_in_demand_techs:
    equations:
    - expression: flow_in
    foreach:
    - nodes
    - techs
    - timesteps
    - carriers
    where: base_tech=demand

  # This alternative option works with the NOT logic moved to the equations where strings 
  flow_out_non_transmission_techs_1:
    where: flow_out
    equations:
      - where: NOT base_tech=transmission
        expression: flow_out
    foreach:
    - nodes
    - techs
    - timesteps
    - carriers

  # Another alternative that works (just manually implementing an OR equivalent)
  flow_out_non_transmission_techs_2:
    equations:
    - where: base_tech=supply
      expression: flow_out
    - where: base_tech=storage
      expression: flow_out
    - where: base_tech=conversion
      expression: flow_out
    foreach:
    - nodes
    - techs
    - timesteps
    - carriers

Which operating systems have you used?

• macOS
• Windows
• Linux

Version

v0.7.0.dev3

Relevant log output

No response

[brynpickering]

So, this is caused by flow_out not having a value for the demand technology group, so it is filled with its default value (the integer 0) in your main example. It isn't an issue of OR/NOT specifically, your other examples do/don't work because they aren't/are leaving those gaps. So, expression: flow_in works with base_tech=demand because all demand technologies have a value for flow_in. Your first non-working example (base_tech=supply OR base_tech=storage OR base_tech=conversion) works fine for me.

Moving NOT base_tech=transmission to the expression where string works because you have where: flow_out at the top level. You'll find that where: flow_out AND NOT base_tech=transmission will work fine at the top level.

Either there should have been an error raised in your main use-case, since the where string effectively left NaNs in the array (for demand techs), or it should have happily passed the unfiltered empty data as zeros along all the way to the end. I can't decide which would be the better result for the user...