A family of sets in $[n]$ is called an $\ell$-Oddtown if the sizes of all sets are not divisible by $\ell$, but the sizes of pairwise intersections are divisible by $\ell$. The problem was completely solved when $\ell$ is a prime via an elegant linear algebraic method, showing that the family has size at most $n$. However, not much was known for composite numbers. By splitting the family into families correspond to each prime factor of $\ell$, one can show that the number is at most $\omega n$, where $omega$ is the number of prime factors of $\ell$. We used both combinatorial and Fourier analytic arguments to prove that the number of sets in any $\ell$-Oddtown is at most $\omega n-(2\omega+\varepsilon)\log_2 n$ for most $n,\ell$.