transform: luaxform transform script

doc/userguide/configuration/suricata-yaml.rst

@@ -2825,8 +2825,8 @@ Lua
 ~~~
 
 Suricata 8.0 sandboxes Lua rules by default. The restrictions on the sandbox for Lua rules can be
-modified in the ``security.lua`` section of the configuration file.  Additionally, Lua rules 
-can be completely disabled the same as the Suricata 7.0 default:
+modified in the ``security.lua`` section of the configuration file.  This section also applies to
+Lua transforms Additionally, Lua rules can be completely disabled the same as the Suricata 7.0 default:
 
 ::
 

doc/userguide/lua/lua-functions.rst

@@ -6,7 +6,8 @@ Lua functions
 Differences between `output` and `detect`:
 ------------------------------------------
 
-Currently, the ``needs`` key initialization varies, depending on what is the goal of the script: output or detection.
+Currently, the ``needs`` key initialization varies, depending on what is the goal of the script: output or detection. The
+Lua script for the ``luaxform`` transform **does not use ``needs``**.
 
 If the script is for detection, the ``needs`` initialization should be as seen in the example below (see :ref:`lua-detection` for a complete example of a detection script):
 
@@ -812,7 +813,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGet
 ~~~~~~~~~~~~~~
 
@@ -828,7 +829,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGetString
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -998,7 +999,7 @@ index so in our case we need to use 0.
          SCFlowintSet(0, a + 1)
      else
          SCFlowintSet(0, 1)
-     end 
+     end
 
 SCFlowintGet
 ~~~~~~~~~~~~
@@ -1031,7 +1032,7 @@ SCFlowvarSet
 Set a Flowvar. First parameter is the index, second is the data
 and third is the length of data.
 
-You can use it to set string 
+You can use it to set string
 
 ::
 
@@ -1041,7 +1042,7 @@ You can use it to set string
      needs["flowvar"] = {"cnt"}
      return needs
  end
- 
+
  function match(args)
      a = SCFlowvarGet(0);
      if a then
@@ -1050,7 +1051,7 @@ You can use it to set string
      else
          a = tostring(1)
          SCFlowvarSet(0, a, #a)
-     end 
+     end
 
 Misc
 ----

doc/userguide/lua/lua-usage.rst

@@ -1,20 +1,27 @@
 Lua usage in Suricata
 =====================
 
-Lua scripting can be used in two components of Suricata. The first is in
-output and the second one in rules in the detection engine.
+Lua scripting can be used in two components of Suricata:
+
+  * Output
+  * Detection: ``lua`` keyword and ``luaxform`` transform
 
 Both features are using a list of functions to access the data extracted by
 Suricata. You can get the list of functions in the :ref:`lua-functions` page.
 
-.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function, depending on what is the usage: ``output`` or ``detection``. The list of available functions may also differ.
+.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function, depending on what is the usage: ``output`` or ``detection``. The list of available functions may also differ. The ``luaxform`` doesn't use the ``needs`` key.
 
 Lua output
 ----------
 
-Lua can be used to write arbitrary output. See :ref:`lua-output` for more information.
+Lua scripts can be used to write arbitrary output. See :ref:`lua-output` for more information.
 
 Lua detection
 -------------
 
-Lua script can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+Lua scripts can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+
+Lua transform
+-------------
+
+The ``luaxform`` transform can be used in signatures.  See :ref:`lua-transform` for more information.

doc/userguide/rules/lua-detection.rst

@@ -3,10 +3,18 @@
 Lua Scripting for Detection
 ===========================
 
+There are 2 ways that Lua can be used with detection. These are
+
+* ``lua`` rule keyword.
+* ``luaxform`` transform.
+
 .. note:: Lua is disabled by default for use in rules, it must be
           enabled in the configuration file. See the ``security.lua``
           section of ``suricata.yaml`` and enable ``allow-rules``.
 
+Lua Rule Keyword
+^^^^^^^^^^^^^^^^
+
 Syntax:
 
 ::
@@ -103,8 +111,13 @@ Entire script:
 
   return 0
 
-Sandbox and Available functions
--------------------------------
+Lua Transform: ``luaxform``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+More details in :ref:`lua-transform`
+
+Lua Sandbox and Available functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Lua rule scripts are run in a sandbox environment the applies the
 following restrictions:
@@ -140,7 +153,7 @@ Of note, the following standard libraries are not available:
 
 This behavior can be modified via the ``security.lua`` section of :ref:`suricata-yaml-lua-config`
 
-.. note:: Suricata 8.0 has moved to Lua 5.4 and has builtin support for bitwise and utf8 operations now.
+.. note:: Suricata 8.0 has moved to Lua 5.4 and now has builtin support for bitwise and utf8 operations.
 
 A comprehensive list of existing lua functions - with examples - can
 be found at :ref:`lua-functions` (some of them, however, work only for

doc/userguide/rules/transforms.rst

@@ -243,3 +243,107 @@ This example transforms `"Zm 9v Ym Fy"` to `"foobar"`::
 
        content:"/?arg=Zm 9v Ym Fy"; from_base64: offset 6, mode rfc2045; \
        content:"foobar";
+
+.. _lua-transform:
+
+luaxform
+--------
+
+This transform allows a Lua script to apply a transformation
+to a buffer.
+
+Lua scripts that are used for transformations *must* contain a function
+named ``transform``.
+
+Lua transforms can be passed optional arguments -- see the examples below -- but they
+are not required to do so. Optional arguments are included with the transform (see example
+below).
+
+Note that the arguments and values are passed without validation
+nor interpretation. There is a maximum of 10 arguments.
+
+The Lua ``transform`` function receive parameters:
+
+  * `input-length` The number of bytes in the buffer provided to the transform
+  * `input` The buffer provided to the transform
+  * `argument` The number of arguments provided in the following parameters. If there are
+    no arguments to the Lua transform, this value will be `0`.
+  * `arguments` The list of arguments.
+
+The Lua ``transform`` must return two values
+
+  * `transformed-buffer` The buffer with the transformed bytes.
+  * `transformed-buffer-byte-count` The number of bytes in the transformed buffer.
+
+This example supplies the HTTP data to a Lua transform with the transform
+results being checked with `content`.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established;  \
+    file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 2;)
+
+
+This example supplies the HTTP data to a Lua transform with the transform
+with arguments that specify the offset and byte count for the transform.
+The resulting buffer is then checked with a `content` match.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established; \
+    file.data; luaxform:./lua/lua-transform.lua,  bytes 12, offset 13; content: "abc"; sid: 1;)
+
+The following Lua script shows a transform that handles arguments: `bytes` and `offset` and uses
+those values (or defaults, if there are no arguments) for applying the uppercase transform to
+the buffer.
+
+.. code-block:: lua
+
+   function init (args)
+        local needs = {}
+        return needs
+   end
+
+   local function get_value(item, key)
+       if string.find(item, key) then
+           local _, value = string.match(item, "(%a+)%s*(%d*)")
+           if value ~= "" then
+               return tonumber(value)
+           end
+       end
+
+       return nil
+   end
+
+   -- Arguments supported
+   local bytes_key = "bytes"
+   local offset_key = "offset"
+   function transform(input_len, input, argc, args)
+       local bytes = input_len
+       local offset = 0
+
+       -- Look for optional bytes and offset arguments
+       for i, item in ipairs(args) do
+           local value = get_value(item, bytes_key)
+           if value ~= nil then
+               bytes = value
+           else
+               local value = get_value(item, offset_key)
+               if value ~= nil then
+                   offset = value
+               end
+           end
+       end
+       local str_len = #input
+       if offset < 0 or offset > str_len then
+           print("offset is out of bounds: " .. offset)
+           return nil
+       end
+       str_len = str_len - offset
+       if bytes < 0 or bytes > str_len then
+           print("invalid bytes " ..  bytes .. " or bytes > length " .. bytes .. " length " .. str_len)
+           return nil
+       end
+       local sub = string.sub(input, offset + 1, offset + bytes)
+       return string.upper(sub), bytes
+   end

doc/userguide/upgrade.rst

@@ -77,6 +77,7 @@ Major changes
     - sip.to
     - sip.content_type
     - sip.content_length
+- New transform ``luaxform`` that uses a Lua script for sticky buffer transformation.  More details in :ref:`lua-transform`
 
 Removals
 ~~~~~~~~

src/Makefile.am

@@ -311,6 +311,7 @@ noinst_HEADERS = \
 	detect-transform-compress-whitespace.h \
 	detect-transform-dotprefix.h \
 	detect-transform-header-lowercase.h \
+	detect-transform-luaxform.h \
 	detect-transform-md5.h \
 	detect-transform-pcrexform.h \
 	detect-transform-sha1.h \
@@ -885,6 +886,7 @@ libsuricata_c_a_SOURCES = \
 	detect-transform-compress-whitespace.c \
 	detect-transform-dotprefix.c \
 	detect-transform-header-lowercase.c \
+	detect-transform-luaxform.c \
 	detect-transform-md5.c \
 	detect-transform-pcrexform.c \
 	detect-transform-sha1.c \

src/detect-dns-answer-name.c

@@ -61,10 +61,10 @@ static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx,
     uint32_t data_len = 0;
 
     if (!SCDnsTxGetAnswerName(txv, to_client, index, &data, &data_len)) {
-        InspectionBufferSetupMultiEmpty(buffer);
+        InspectionBufferSetupMultiEmpty(det_ctx, buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
     return buffer;
 }

src/detect-dns-query-name.c

@@ -61,10 +61,10 @@ static InspectionBuffer *GetBuffer(DetectEngineThreadCtx *det_ctx,
     uint32_t data_len = 0;
 
     if (!SCDnsTxGetQueryName(txv, to_client, index, &data, &data_len)) {
-        InspectionBufferSetupMultiEmpty(buffer);
+        InspectionBufferSetupMultiEmpty(det_ctx, buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
     return buffer;
 }

src/detect-dns-query.c

@@ -82,10 +82,10 @@ static InspectionBuffer *DnsQueryGetData(DetectEngineThreadCtx *det_ctx,
     const uint8_t *data;
     uint32_t data_len;
     if (SCDnsTxGetQueryName(txv, false, local_id, &data, &data_len) == 0) {
-        InspectionBufferSetupMultiEmpty(buffer);
+        InspectionBufferSetupMultiEmpty(det_ctx, buffer);
         return NULL;
     }
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->flags = DETECT_CI_FLAGS_SINGLE;
 
     SCReturnPtr(buffer, "InspectionBuffer");

src/detect-engine-frame.c

@@ -67,8 +67,8 @@ static bool SetupStreamCallbackData(struct FrameStreamData *dst, const TcpSessio
 
 static bool BufferSetup(struct FrameStreamData *fsd, InspectionBuffer *buffer, const uint8_t *input,
         const uint32_t input_len, const uint64_t input_offset);
-static void BufferSetupUdp(InspectionBuffer *buffer, const Frame *frame, const Packet *p,
-        const DetectEngineTransforms *transforms);
+static void BufferSetupUdp(DetectEngineThreadCtx *det_ctx, InspectionBuffer *buffer,
+        const Frame *frame, const Packet *p, const DetectEngineTransforms *transforms);
 
 void DetectRunPrefilterFrame(DetectEngineThreadCtx *det_ctx, const SigGroupHead *sgh, Packet *p,
         const Frames *frames, const Frame *frame, const AppProto alproto)
@@ -159,7 +159,7 @@ static void PrefilterMpmFrame(DetectEngineThreadCtx *det_ctx, const void *pectx,
         if (frame->offset >= p->payload_len)
             return;
 
-        BufferSetupUdp(buffer, frame, p, ctx->transforms);
+        BufferSetupUdp(det_ctx, buffer, frame, p, ctx->transforms);
         const uint32_t data_len = buffer->inspect_len;
         const uint8_t *data = buffer->inspect;
 
@@ -251,8 +251,8 @@ bool DetectRunFrameInspectRule(ThreadVars *tv, DetectEngineThreadCtx *det_ctx, c
     return false;
 }
 
-static void BufferSetupUdp(InspectionBuffer *buffer, const Frame *frame, const Packet *p,
-        const DetectEngineTransforms *transforms)
+static void BufferSetupUdp(DetectEngineThreadCtx *det_ctx, InspectionBuffer *buffer,
+        const Frame *frame, const Packet *p, const DetectEngineTransforms *transforms)
 {
     uint8_t ci_flags = DETECT_CI_FLAGS_START;
     uint32_t frame_len;
@@ -275,7 +275,7 @@ static void BufferSetupUdp(InspectionBuffer *buffer, const Frame *frame, const P
             AppLayerParserGetFrameNameById(p->flow->proto, p->flow->alproto, frame->type),
             frame->offset, frame->type, frame->len);
 
-    InspectionBufferSetupMulti(buffer, transforms, data, data_len);
+    InspectionBufferSetupMulti(det_ctx, buffer, transforms, data, data_len);
     buffer->inspect_offset = 0;
     buffer->flags = ci_flags;
 }
@@ -301,7 +301,7 @@ static int DetectFrameInspectUdp(DetectEngineThreadCtx *det_ctx,
         return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
 
     if (!buffer->initialized)
-        BufferSetupUdp(buffer, frame, p, transforms);
+        BufferSetupUdp(det_ctx, buffer, frame, p, transforms);
     DEBUG_VALIDATE_BUG_ON(!buffer->initialized);
     if (buffer->inspect == NULL)
         return DETECT_ENGINE_INSPECT_SIG_NO_MATCH;
@@ -361,7 +361,7 @@ static bool BufferSetup(struct FrameStreamData *fsd, InspectionBuffer *buffer, c
             if (fo_inspect_offset >= (uint64_t)frame->len) {
                 SCLogDebug("data entirely past frame (%" PRIu64 " > %" PRIi64 ")",
                         fo_inspect_offset, frame->len);
-                InspectionBufferSetupMultiEmpty(buffer);
+                InspectionBufferSetupMultiEmpty(fsd->det_ctx, buffer);
                 return false;
             }
 
@@ -387,7 +387,7 @@ static bool BufferSetup(struct FrameStreamData *fsd, InspectionBuffer *buffer, c
     }
     // PrintRawDataFp(stdout, data, data_len);
     SCLogDebug("fsd->transforms %p", fsd->transforms);
-    InspectionBufferSetupMulti(buffer, fsd->transforms, data, data_len);
+    InspectionBufferSetupMulti(fsd->det_ctx, buffer, fsd->transforms, data, data_len);
     SCLogDebug("inspect_offset %" PRIu64, fo_inspect_offset);
     buffer->inspect_offset = fo_inspect_offset;
     buffer->flags = ci_flags;