transform: luaxform transform script

doc/userguide/configuration/suricata-yaml.rst

@@ -2829,8 +2829,9 @@ Lua
 ~~~
 
 Suricata 8.0 sandboxes Lua rules by default. The restrictions on the sandbox for Lua rules can be
-modified in the ``security.lua`` section of the configuration file.  Additionally, Lua rules 
-can be completely disabled the same as the Suricata 7.0 default:
+modified in the ``security.lua`` section of the configuration file.  This section also applies to
+Lua transforms. Additionally, Lua rules can be completely disabled in the same way as for as the
+Suricata 7.0 default:
 
 ::
 

doc/userguide/lua/lua-functions.rst

@@ -6,7 +6,8 @@ Lua functions
 Differences between `output` and `detect`:
 ------------------------------------------
 
-Currently, the ``needs`` key initialization varies, depending on what is the goal of the script: output or detection.
+Currently, the ``needs`` key initialization varies, depending on what is the goal of the script: output or detection. The
+Lua script for the ``luaxform`` transform **does not use ``needs``**.
 
 If the script is for detection, the ``needs`` initialization should be as seen in the example below (see :ref:`lua-detection` for a complete example of a detection script):
 
@@ -812,7 +813,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGet
 ~~~~~~~~~~~~~~
 
@@ -828,7 +829,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGetString
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -998,7 +999,7 @@ index so in our case we need to use 0.
          SCFlowintSet(0, a + 1)
      else
          SCFlowintSet(0, 1)
-     end 
+     end
 
 SCFlowintGet
 ~~~~~~~~~~~~
@@ -1031,7 +1032,7 @@ SCFlowvarSet
 Set a Flowvar. First parameter is the index, second is the data
 and third is the length of data.
 
-You can use it to set string 
+You can use it to set string
 
 ::
 
@@ -1041,7 +1042,7 @@ You can use it to set string
      needs["flowvar"] = {"cnt"}
      return needs
  end
- 
+
  function match(args)
      a = SCFlowvarGet(0);
      if a then
@@ -1050,7 +1051,7 @@ You can use it to set string
      else
          a = tostring(1)
          SCFlowvarSet(0, a, #a)
-     end 
+     end
 
 Misc
 ----

doc/userguide/lua/lua-usage.rst

@@ -1,20 +1,29 @@
 Lua usage in Suricata
 =====================
 
-Lua scripting can be used in two components of Suricata. The first is in
-output and the second one in rules in the detection engine.
+Lua scripting can be used in two components of Suricata:
+
+  * Output
+  * Detection: ``lua`` keyword and ``luaxform`` transform
 
 Both features are using a list of functions to access the data extracted by
 Suricata. You can get the list of functions in the :ref:`lua-functions` page.
 
-.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function, depending on what is the usage: ``output`` or ``detection``. The list of available functions may also differ.
+.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function,
+   depending on what is the usage: ``output`` or ``detection``. The list of available
+   functions may also differ. The ``luaxform`` doesn't use the ``needs`` key.
 
 Lua output
 ----------
 
-Lua can be used to write arbitrary output. See :ref:`lua-output` for more information.
+Lua scripts can be used to write arbitrary output. See :ref:`lua-output` for more information.
 
 Lua detection
 -------------
 
-Lua script can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+Lua scripts can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+
+Lua transform
+-------------
+
+The ``luaxform`` transform can be used in signatures.  See :ref:`lua-transform` for more information.

doc/userguide/rules/lua-detection.rst

@@ -3,10 +3,18 @@
 Lua Scripting for Detection
 ===========================
 
+There are 2 ways that Lua can be used with detection. These are
+
+* ``lua`` rule keyword.
+* ``luaxform`` transform.
+
 .. note:: Lua is disabled by default for use in rules, it must be
           enabled in the configuration file. See the ``security.lua``
           section of ``suricata.yaml`` and enable ``allow-rules``.
 
+Lua Rule Keyword
+^^^^^^^^^^^^^^^^
+
 Syntax:
 
 ::
@@ -103,8 +111,13 @@ Entire script:
 
   return 0
 
-Sandbox and Available functions
--------------------------------
+Lua Transform: ``luaxform``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+More details in :ref:`lua-transform`
+
+Lua Sandbox and Available functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Lua rule scripts are run in a sandbox environment the applies the
 following restrictions:
@@ -140,7 +153,7 @@ Of note, the following standard libraries are not available:
 
 This behavior can be modified via the ``security.lua`` section of :ref:`suricata-yaml-lua-config`
 
-.. note:: Suricata 8.0 has moved to Lua 5.4 and has builtin support for bitwise and utf8 operations now.
+.. note:: Suricata 8.0 has moved to Lua 5.4 and now has builtin support for bitwise and utf8 operations.
 
 A comprehensive list of existing lua functions - with examples - can
 be found at :ref:`lua-functions` (some of them, however, work only for

doc/userguide/rules/transforms.rst

@@ -243,3 +243,110 @@ This example transforms `"Zm 9v Ym Fy"` to `"foobar"`::
 
        content:"/?arg=Zm 9v Ym Fy"; from_base64: offset 6, mode rfc2045; \
        content:"foobar";
+
+.. _lua-transform:
+
+luaxform
+--------
+
+This transform allows a Lua script to apply a transformation
+to a buffer.
+
+Lua scripts that are used for transformations *must* contain a function
+named ``transform``.
+
+Lua transforms can be passed optional arguments -- see the examples below -- but they
+are not required to do so. Optional arguments are included with the transform (see example
+below).
+
+Note that the arguments and values are passed without validation
+nor interpretation. There is a maximum of 10 arguments.
+
+The Lua ``transform`` function receive parameters:
+
+  * `input-length` The number of bytes in the buffer provided to the transform
+  * `input` The buffer provided to the transform
+  * `argument` The number of arguments provided in the following parameters. If there are
+    no arguments to the Lua transform, this value will be `0`.
+  * `arguments` The list of arguments.
+
+The Lua ``transform`` must return two values
+
+  * `transformed-buffer` The buffer with the transformed bytes.
+  * `transformed-buffer-byte-count` The number of bytes in the transformed buffer.
+
+This example supplies the HTTP data to a Lua transform with the transform
+results being checked with `content`.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established;  \
+    file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 2;)
+
+
+The following example supplies the HTTP data to a Lua transform script along with
+with arguments that specify the offset and byte count for the transform.
+The resulting buffer is then checked with a `content` match.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established; \
+    file.data; luaxform:./lua/lua-transform.lua, bytes 12, offset 13; content: "abc"; sid: 1;)
+
+The following Lua script shows a transform example that handles arguments: `bytes` and `offset` and uses
+those values (or defaults, if there are no arguments) for applying the uppercase transform to
+the buffer.
+
+.. code-block:: lua
+
+    function init (args)
+    -- Initialization time steps can be
+    -- performed here.
+    end
+
+    local function get_value(item, key)
+       if string.find(item, key) then
+           local _, value = string.match(item, "(%a+)%s*(%d*)")
+           if value ~= "" then
+               return tonumber(value)
+           end
+       end
+
+       return nil
+    end
+
+    -- Arguments supported
+    local bytes_key = "bytes"
+    local offset_key = "offset"
+    function transform(input_len, input, argc, args)
+       local bytes = input_len
+       local offset = 0
+
+       -- Look for optional bytes and offset arguments
+       for i, item in ipairs(args) do
+           local value = get_value(item, bytes_key)
+           if value ~= nil then
+               bytes = value
+           else
+               value = get_value(item, offset_key)
+               if value ~= nil then
+                   offset = value
+               end
+           end
+       end
+
+       local str_len = #input
+       if offset < 0 or offset > str_len then
+           print("offset is out of bounds: " .. offset)
+           return nil
+       end
+
+       local avail_len = str_len - offset
+       if bytes < 0 or bytes > avail_len then
+           print("invalid bytes " ..  bytes .. " or bytes exceeds available  length " .. avail_len)
+           return nil
+       end
+
+       local sub = string.sub(input, offset + 1, offset + bytes)
+       return string.upper(sub), bytes
+    end

doc/userguide/upgrade.rst

@@ -79,6 +79,7 @@ Major changes
     - sip.content_length
 - Napatech support has been moved to a capture plugin. See :doc:`Napatech plugin
   <upgrade/8.0-napatech-plugin>`.
+- New transform ``luaxform`` that uses a Lua script for sticky buffer transformation.  More details in :ref:`lua-transform`
 
 Removals
 ~~~~~~~~

src/Makefile.am

@@ -310,6 +310,7 @@ noinst_HEADERS = \
 	detect-transform-compress-whitespace.h \
 	detect-transform-dotprefix.h \
 	detect-transform-header-lowercase.h \
+	detect-transform-luaxform.h \
 	detect-transform-md5.h \
 	detect-transform-pcrexform.h \
 	detect-transform-sha1.h \
@@ -879,6 +880,7 @@ libsuricata_c_a_SOURCES = \
 	detect-transform-compress-whitespace.c \
 	detect-transform-dotprefix.c \
 	detect-transform-header-lowercase.c \
+	detect-transform-luaxform.c \
 	detect-transform-md5.c \
 	detect-transform-pcrexform.c \
 	detect-transform-sha1.c \

src/detect-dce-stub-data.c

@@ -79,8 +79,8 @@ static InspectionBuffer *GetSMBData(DetectEngineThreadCtx *det_ctx,
             return NULL;
         SCLogDebug("have data!");
 
-        InspectionBufferSetup(det_ctx, list_id, buffer, data, data_len);
-        InspectionBufferApplyTransforms(buffer, transforms);
+        InspectionBufferSetupAndApplyTransforms(
+                det_ctx, list_id, buffer, data, data_len, transforms);
     }
     return buffer;
 }
@@ -105,8 +105,8 @@ static InspectionBuffer *GetDCEData(DetectEngineThreadCtx *det_ctx,
         } else {
             buffer->flags |= DETECT_CI_FLAGS_DCE_BE;
         }
-        InspectionBufferSetup(det_ctx, list_id, buffer, data, data_len);
-        InspectionBufferApplyTransforms(buffer, transforms);
+        InspectionBufferSetupAndApplyTransforms(
+                det_ctx, list_id, buffer, data, data_len, transforms);
     }
     return buffer;
 }

src/detect-dnp3.c

@@ -166,8 +166,8 @@ static InspectionBuffer *GetDNP3Data(DetectEngineThreadCtx *det_ctx,
         }
 
         SCLogDebug("tx %p data %p data_len %u", tx, tx->buffer, tx->buffer_len);
-        InspectionBufferSetup(det_ctx, list_id, buffer, tx->buffer, tx->buffer_len);
-        InspectionBufferApplyTransforms(buffer, transforms);
+        InspectionBufferSetupAndApplyTransforms(
+                det_ctx, list_id, buffer, tx->buffer, tx->buffer_len, transforms);
     }
     return buffer;
 }

src/detect-dns-answer-name.c

@@ -64,7 +64,7 @@ static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx,
         InspectionBufferSetupMultiEmpty(buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
     return buffer;
 }

src/detect-dns-query-name.c

@@ -64,7 +64,7 @@ static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx,
         InspectionBufferSetupMultiEmpty(buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
     return buffer;
 }

src/detect-dns-query.c

@@ -85,7 +85,7 @@ static InspectionBuffer *DnsQueryGetData(DetectEngineThreadCtx *det_ctx,
         InspectionBufferSetupMultiEmpty(buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
 
     SCReturnPtr(buffer, "InspectionBuffer");